Jigsaw-ViT: Learning jigsaw puzzles in vision transformer

The success of Vision Transformer (ViT) in various computer vision tasks has promoted the ever-increasing prevalence this convolution-free network. fact that ViT works on image patches makes it potentially relevant to problem jigsaw puzzle solving, which is a classical self-supervised task aiming at reordering shuffled sequential back their original form. Solving been demonstrated be helpful for diverse using Convolutional Neural Networks (CNNs), such as feature representation learning, domain generalization and fine-grained classification. In paper, we explore solving auxiliary loss classification, named Jigsaw-ViT. We show two modifications can make Jigsaw-ViT superior standard ViT: discarding positional embeddings masking randomly. Yet simple, find proposed able improve both robustness over ViT, usually rather trade-off. Numerical experiments verify adding branch provides better large-scale classification ImageNet. Moreover, also improves against noisy labels Animal-10N, Food-101N, Clothing1M, well adversarial examples. Our implementation available https://yingyichen-cyy.github.io/Jigsaw-ViT.